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ABSTRACT 

A cluster of schools in Montgomery County, Maryland, 
is working to meet the changing educational needs of its student 
population. The state of Maryland has developed partnerships with 
business and local service providers to improve school performance. 
House Bill 874 provided for Challenge School funds to help local 
schools reach Maryland's Schools for Success Goals by the year 2000. 
This paper presents an interim evaluation design that supported 
school-improvement plan (SIP) implementation, identified 
program-related activities and strategies, documented Challenge 
Schools' progress, and identified SIP outcomes. The Challenge School 
funding plan was implemented in 1993-94 in five elementary, one 
middle, and one high school. Quantitative and qualitative data 
collected at the school sites indicate that the program resulted in 
increased parent .involvement and improved school/agency 
responsiveness to children and families. The role of the State 
Department of Education is expected to change from one of 
advisor/monitor to observer as schools take greater responsibility 
for self-evaluation. (LMl) 
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An Evaluation Design for a Community School Cluster 
Antoinette Brown, Ph. D. 

Background 

A cluster of schools in Montgomery County, Maryland is working to 
meet the changing educational demands of its student population. The 
State of Maryland, the schools' main partner through Challenge Grant 
funds, has combined forces with businesses and local service providers to 
forge a school/agency cooperative effort. Together with parents and other 
community members, their efforts enable the schools to offer students 
and parents a variety of services to meet pressing needs. 

The State of Maryland's House Bit! Number 874 provided for 
Challenge Schools funds to support local schools in reaching Schools for 
Success Goals by the year 2000. Challenge Grant funding is intended to 
improve the schools' performance in terms of variables reported in the 
annual report card issued by the Maryland State Department of Education. 
Key variables include the following: attendance rates; drop-out rates; 
promotion rates; passing rates on functional tests in math, reading, 
writing and citizenship; and student achievement on assessment tests 
administered in grades 3, 5 and 8. 

The Challenge Grant was authorized for three school years, from 
1992-93 to 1994-95. In August of 1992, Montgomery County Public 
Schools (MCPS) selected a cluster of five elementary schools, a middle 
school and a high school to develop and implement school improvement 
plans designed to address the needs of a high mobility and high-proportion 
ESOL population. 

The first year (1992-1993) was a planning year during which each 
school improvement team (SIT) examined its school's needs and resources 
and developed an appropriate school improvement plan (SIP) with 
identifiable goals, objectives, implementation strategies and measureable 
indicators of progress toward the goals. The first year of implementation 
was 1993-94. 

Statement of Purpose 

Evaluation of school-based services can provide information about 
the effectiveness of the services, and also improve the delivery of 
services (Gomby and Larson, 1992). In the context of the Challenge Grant, 



there were multiple clients for evaluation information, and each client 
had different points of view about the goals of the evaluation, and of the 
services provided by the Challenge program. 

Among the various clients were the local school community, the 
county school system, the state department of education, and the state 
legislature. Because of the multiple clients and goals in this endeavor, 
the MCPS Department of Educational Accountability (DEA) explored, with 
the schools, an evaluation design that included a range of evaluation 
strategies, both formative and summative. 

The design that emerged was determined by the school-community 
context in which the Challenge Grant was implemented and by the data 
needs of the school, the local school system and the state department of 
education. The process (formative) evaluation described the program 
activities, how the programs were implemented, who was involved, 
problems that arose, and successes that resulted. The outcome 
(summative) evaluation measured the progress, toward established 
measureable goals and objectives, particularly those of the individual 
Challenge Grant School Improvement Plans (SIPs), the Montgomery County 
Success for Every Student goals, and the Maryland State Schools for 
Success goals. 

The paper presents an interim evaluation design that supported the 
implementation of the SIPs, identified program-related activities and 
stratgies, documented Challenge schools' progress, and identified 
outcomes of the SIPs' activities. 

Conceptual Framework 

The long-range goal of the Challenge Grant evaluation was to 
contribute to the effectiveness of grant-funded activities and to enhance 
positive outcomes. DEA, as the unit conducting the evaluation, became an 
actor in the cooperative school/community effort rather than just an 
external evaluator. Using the "learning community" metaphor (St. John, 
1992) DEA joined a group of people, all playing different roles, and all 
involved in learning together how to improve the outcomes of the 
Challenge Grant activities. 

Relationships between DEA and other actors have been collegial 
rather than adversarial and roles flexible. DEA has retained an 
inc" pendent perspective, while at the same time, it has played a number 
of roles. Within the learning communitv DEA has acted as a collaborator, 
an educator, a learner, a mirror, a recorder, an advisor, and occasionally 



as a critic. 

St. John (1992) proposed three functions of evaluation within a 
learning community: to import information into the community; to create 
opportunities for interaction within the community; and to export 
information out of the community so that others can learn from the effort. 
For the first year, DEA's functions have concentrated on importing 
information into the learning community and to a lesser extent, exporting 
information out of the learning community. 

B ackground 

The State of Maryland's House Bill 874 provided for Challenge School 
funds to support local schools to reach Schools for Success Goals by the 
year 2000. The Challenge Schools program was designed to bring 
systematic change to low-performing schools through strategic planning, 
direct intervention, expanded opportunities, and resources - both human 
and financial. As a result of House Bill 874, twenty-nine schools in 
Prince George's County, Montgomery County, and Baltimore City school 
systems began a collaborative effort that requires school improvement 
teams (SITs) at each school to develop school improvement plans (SIPS). 
SIPs include parents, teachers, administrators, supporting service 
employees, business partners, and community members. 

Written agreements were negotiated between the state and local 
superintendent and between the local superintendent and the school 
principal. The agreements focused funding and action on the school 
principal, SIT, and instructional program to improve school performance 
as measured against Maryland School Performance goals.The county's 
intention was that these seven schools would provide models of 
successful programs addressing the issues of high mobility and a large 
ESOL population. 

From the school community's point of view, the aim of the 
school/agency cooperation was to make the school a center for community 
involvement in activities that can foster learning. The extra support 
services offered by the seven schools cost about $1,900,000 for the 1993 
1994 school year. That amounts to only about $400 /child above the 
average $7,377 that Montgomery County spent in FY 93, but less than it 
would cost to provide these services at separate support centers. 



The Target Population 



The seven schools in the cluster share a number of demographic 
characteristics including racial and ethnic diversity, a high percentage of 
students with limited English proficiency, high student mobility, and a 
high percentage of students eligible for free or reduced price meals. 
African American, Asian American, and Hispanic students make up about 
60% of the students in the cluster schools. The percentage ranges from 
48% at ES 807 to 78% at ES 788. 

In 1993 the percentage of limited English proficient students ranged 
from 6% at ES 807 to 12% at the high school. Students in the cluster with 
limited English proficiency tend to score below the cluster average on the 
state's functional tests. Because passage of these tests is required by the 
state for high school graduation, the low test scores are a deterrent for 
these students to earn a Maryland high school diploma. 

The cluster schools have high student mobility rates for all ethnic 
and racial groups. The school mobility rate (entrants and withdrawals 
during the school year) in 1993 was over 30% in several schools within 
the cluster. About one-third of the students enrolled in the cluster are 
eligible for free or reduced price meals. The percentage ranged from 24% 
at ES 807 to 61% at ES 797. 

School Improvement Pla ns 

During the 1992-1993 school year, each Challenge school identified 
evidence of needs within the school community and prepared a school 
improvement plan that addressed those needs through specific strategies 
and established measureable goals and objectives. 

Several common themes appeared in the goals and objectives 
presented by the seven schools in their SIPs. The first of these themes is 
the improvement of student achievement as reflected in better report card 
grades and higher scores on the Maryland School performance Assessment 
Program, with a focus on the achievement of students enrolled in English 
for Speakers of Other Languages (ESOL) classes. Common strategies to 
support this goal are the use of computer technology and greater parent 
involvement. 

A second recurrent theme is the facilitation of the transition and 
adaptation of new students. Among those targeted are ninth grade 
students entering the high school and transfer students. A third theme is 
the improvement of the home/school connection, usually through improved 



school outreach to parents. A number of schools have targeted parents of 
ESOL students and Limited English Proficient parents in particular. The 
fourth of the common themes is staff development, particularly to support 
improved student achievement. 

Partners in the Challenge Grant effort have taken steps to reach 
their goals by adopting new initiatives and coordinating their activities. 
Four of the elementary schools coordinated a summer math pronram at one 
site to meet the needs of their lowest-performing students. 

Middle School 812, which serves about 1,000 students in grades 6 
through 8, offered extended day programs after regular classes and a 
summer camp. The elementary schools in the cluster offered Head Start 
classes, after-school programs, family resource centers, and services 
targeted to the ESOL community. 

To draw older students and their parents, the high school offered an 
after school and evening program that includes an open study hall and 
computer lab, evening appointments for parents to meet with guidance 
counselors, and SAT/PSAT/TOEFL/CELT preparation. The teens also 
perform community service and tutor younger children at Wheaton cluster 
elementary schools. 

While meeting school and community goals, efforts are also paying 
off in student performance outcomes in both reading and math and higher 
average student attendance rates. 

Evaluation Design 1993-1994 

The 1993-1994 evaluation design emerged from a collaborative 
process between DEA, the Challenge Grant facilitator, and representatives 
of the seven schools. The design addressed the need for both process and 
outcome evaluation at the program level, at the school site, and at the 
cluster level. The process evaluation was primarily school-specific. Each 
of the Challenge Schools was evaluated on the progress it made towards 
implementing its own SIP and on the outcomes resulting from that plan. 

Program-specific data was collected for the purpose of improving 
specific programs and for identifying those programs that have the 
possibility of being replicated at other MCPS school. 

The collected data also had to meet the needs of the various actors: 
state, county, service provider, and school community. The schools and 
service providers are interested in demonstrating that they are meeting 
the needs of the community. Schools must also submit evaluations to 
MSDE to show progress towards meeting school and state goals. The 
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legislature must be able to justify the expenditure of public funds. MSDE 
and the county require evidence that the schools are progressing toward 
state and local goals. 

Methods 

Both quantitative and qualitative data were collected and analyzed 
for the process and outcome evaluation. Quantitative data collected 
routinely by the state, county and schools were supplemented by 
quantitative and qualitative data collected at the school sites. 

The individual school improvement plans specified, for each goal, 
the desired outcomes, and the data that would be collected to document 
the outcomes (milestones). The plans specified that such milestones were 
to be collected quarterly. The approach adopted at the school sites was to 
work with the SITs to identify their data needs and to involve them in the 
design and implementation of the monitoring process. 

The quantitative data methods included the analysis of routinely 
collected data such as enrollment, attendance, student mobility, 
promotion, drop out rate, county criteria-referenced test scores, Maryland 
School Performance Assessment Program results, SAT scores, PSAT 
scores, and report card grades. 

Quantitative data was used to describe school characteristics 
(enrollment, numbers of students receiving special services, etc. ) and to 
document outcomes that might be attributable to Challenge Grant projects 
(changes in SAT and PSAT scores, improvements in attendance, reduction 
of suspensions, etc.). 

Schools also undertook site-specific studies about programs at that 
school. At some schools this involved the analysis of data such as 
discipline reports or school-administered tests. In other cases, schools, 
with assistance from DEA, designed, administered, and analyzed surveys 
that answered questions about Challenge Grant activities. Surveys were 
distributed to parents and teachers, and at the high school level, to 
students. 

The quantitative data was reported in a series of school-specific 
tables that summarized school years 1992-1993 and 1993-1994. Tables 
were organized into four categories: student population characteristics 
(enrollment, student mobility, etc.); student participation (attendance 
rates); student attainment (promotion rates); and assessed student 
knowledge (test results). Data was generally presented for the School as 
a whole, and also disaggregated by sex and race/ethnicity to clarify the 
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impact of the programs on subpopulations. 

A wide variety of qualitative data contributed to the process and 
outcome evaluation including DEA observations, principal observations, 
document reviews, interviews, and student portfolio assessment. 
Observations were made at each of the Challenge schools. An observer 
attended SIT meetings, team training provided by MSDE, and other Grant- 
related activities. 

The qualitative data was reported in a narrative case report form 
that was roughly similar for each school and for the cluster as a whole. 
Drafts of the case studies were distributed for member checking to the 
SITs and other persons who provided information. Responses were 
incorporated into the narratives to produce a "negotiated outcome" 
(Lincoln & Guba, 1985 , p. 211). 

The original design stipulated the completion of two annual reports 
and two interim reports using a standard format to report narrative and 
quantitative information about each of the seven schools. The reporting 
design was modified frequently to meet the information needs of the 
multiple clients. Report formats during the first year have been as varied 
as was the data. They have included responses to telephone requests for 
information, a formal cluster-level report combining narrative and 
quantitative information, and school-specific evaluations required by the 
State of Maryland. 

Roles in the Learning Community 

DEA's function within the learning community during the first year 
ha been primarily to import information into the community. 
DEA served this function by filling the role of an advisor providing 
technical assistance to other members of the learning community, 
particularly the schools. DEA assisted SITs with the development, 
administration and analysis of surveys, analyzed test results, and helped 
to design monitoring schemes. DEA worked with the schools to develop 
self-assessments that were required by the MSDE and that served as the 
basis of subsequent SiPs. The importing function will probably continue 
to be significant during the 1994-1995 school year. 

A secondary function during the first year was to export 
information. This was accomplished by acting as a reporter to a number 
of clients, including the schools themselves, to the local and State Boards 
of Education, to MSDE, and to the State legislature. The information that 
has been reported has addressed primarily the process of implementing 
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grant-funded activities and less about outcomes of the process. During 
the second year, it is anticipated that the reporting function will take on 
more importance. 

The function of creating opportunities for interactions within the 
community has not been explored much. Only one formal site visit was 
conducted by MSDE, although informal visits were made to Challenge 
schools by MSDE staff members. The seven school principals meet 
together regularly to coordinate their activities, but the only 
opportunities for the SITs to meet together have been during training 
conducted by MSDE. Additional agencies and business partners have joined 
the community in the second year of implementation. Exploring ways for 
participants to interact will be a priority for the learning community in 
the coming year. 

Findings and Implications 

Proponents of school-linked services have identified three elements 
that are required to achieve positive outcomes: comprehensive services, 
increased parental involvement, and responsiveness of schools and 
agencies to children and families (Gomby and Larson, 1992). 

Our process evaluation documents the existence of at least two of 
these elements in Challenge schools: increased parental involvement and 
improved responsiveness of schools and agencies to children and families. 
The comprehensiveness of services provided to children and families does 
not now meet all the needs identified by the school communities and 
school improvement teams. Nevertheless, the preliminary summative 
evaluation has shown that there have been positive outcomes in terms of 
student attainment and achievement, and that there are successful 
programs with the potential for replicability. 

In the last year of the Challenge Grant, the evaluation plan and 
methodology is evolving to meet the needs of the seven cluster schools 
and to meet the changing criteria imposed by the State of Maryland. 
During the first year of implementation, the process evaluation focused on 
documenting the history of the Challenge program as well as 
implementation. During the second year of implementation, the focus will 
shift slightly to emphasize the identification of successful programs with 
potential for replicability. 

The roles of the various members of the learning community are also 
changing, including that of DEA. In the first year of implementation, DEA 
frequently acted as an advisor and educator. The original approach of 
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involving participants in the monitoring process required DEA to devote 
time to educating school staff about monitor'ng and evaluation procedures 
and assisting the SITs to develop monitoring schemes for self-evaluation. 
During the second year, it is expected that school staff will take on more 
responsibility for self-evaluation, allowing DEA to spend less time 
advising, and more time observing and reporting. 
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